Deriving Musical Control Features from a Real-Time

A novel system that analyzes timbral information of a live clarinet performance for expressive musical gesture is presented. This work is motivated by the growing need to accurately describe the subtle gestures and nuances of single-line acoustic instrument performances. MIDI has successfully satisfied this need for percussion-type instruments, such as the piano. However, a method for extracting gesture of other instruments, and a MIDI-like protocol by which computers may communicate about these gestures, has been lacking. The exploration of timbre attempts to fulfill the goal of deriving such a parameter stream from the clarinet. Timbre is a particularly evasive attribute of instrumental sound, yet one that is intimately tied to musical expression. This system uses Principle Component Analysis, a supervised pattern recognition technique, to extract musical gesture parameters based upon the timbre of an acoustic signal. These parametric descriptors can then be used as a control mechanism for interactive music systems, and as a means of deriving high level musical expression. Thesis Supervisor: Tod Machover Title: Associate Professor of Music and Media, MIT Media Laboratory This work is funded in part by the Yamaha Corporation and Sega of America. Acknowledgments Of all the people at the Lab who helped shape this thesis, I owe the most gratitude to Eric M6tois. His generous advice, pertinent ideas, and technical expertise were invaluable. Damon Horowitz, Eric, and Dan Ellis were exceptionally helpful by providing comments and corrections after sifting through my drafts. Dan was particularly encouraging at a time when I was losing focus and needed a boost. Michael Casey gave me the first glimpse of pattern recognition and helped direct my initial ideas. The Media Lab experience would not have been fulfilling without the friendship and talent of my "partners in media": Ed Hammond, Eric "EMJ" Jordan, Teresa Marrin, Fumi Matsumoto, Suzanne McDermott, Eric M6tois, Joe Paradiso, Pete "Phat" Rice, Josh Smith, David Waxman, Mike Wu, and John Underkoffler. In particular, my officemates Alex Rigopulos and Damon Horowitz have been an endless source of comedy, provocative ideas, support, and puerile behavior. I owe thanks to Michael Hawley for first showing me the lab when I was a UROP and exposing me to the world of media and music. Thanks to Aba and Ima for giving me the opportunities to grow and for being an enormous source of support. Offer, myfavorite brother, deserves special thanks for being a great friend and for putting up with me for so long. This thesis would not have been possible without the generous musical training I have received throughout the past twelve years. Thanks to my clarinet teachers Louise Goni, Jonathan Cohler, and Bill Wrzecien. Years of playing chamber music with Elaine Chew, Wilson Hsieh, Julia Ogrydziak, and Donald Yeung, among the many others, has taught me how to become a better musician and has even made MIT bearable. Finally, I'd like to warmly thank my advisor Tod Machover for his extraordinary support, amazing ideas, and encouraging advice. The unique environment he has set up for us at the Media Lab has allowed me to develop my interests, and explore this exciting field.